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ABSTRACT 


In August 2012, Naval Medical Center San Diego implemented a state-of-the-art 
pharmacy automation system in an effort to reduce cost and improve efficiency. The 
objective of this study is to quantify the increase in efficiency after installation through a 
focus on observed post-automation prescription fill times during calendar year 2014 
(CY2014) and a simulated pre-automation process. With a response of average daily 
prescription fill time, automatic prescription fills in CY2014 are quicker than manual 
prescription fills in CY2014 by 6.97 ±0.97 (standard error) minutes, and post-automation 
prescription fills are quicker than pre-automation prescription fills by 4.4 ±0.34 minutes. 
The difference between pre-automation and post-automation prescription fills is used as 
the response in a linear regression to determine which factors most contribute to the 
decrease in prescription fill time. The proportion of prescriptions automated is influential: 
if this proportion is held constant at 0.37, the workload for each pharmacy technician can 
be reduced by an estimated 2.34 ±0.03 (standard deviation) hours per day. A cost analysis 
of the pharmacy automation system is conducted, and it is estimated that a lower bound 
on the annual cost savings after implementation is over $300,000. 
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EXECUTIVE SUMMARY 


In today’s healthcare environment, healthcare facilities continue to seek ways to 
reduce cost and improve efficiency; the pharmacy is one of many avenues to accomplish 
this task. In 2012, Naval Medical Center San Diego (NMCSD) installed a state-of-the-art 
pharmacy automaton system in an attempt to increase efficiency within their pharmacy. 

Prior to 2012, pharmacy technicians manually filled all prescription orders within 
the NMCSD pharmacy. The new pharmacy automation system is capable of 
autonomously completing all the steps in the prescription fill process prior to pharmacist 
verification. While the automation system at NMCSD does not completely eliminate the 
need for manual prescription fills, it has the potential to greatly reduce the workload 
placed on pharmacy technicians. The objective of this study is to quantify the increases in 
efficiency experienced after the installation of pharmacy automation at NMCSD. 

Of interest is the total prescription fill time for each medication, which is the time 
elapsed between when a customer orders the prescription at the intake window and the 
time pharmacist verification is completed. Analysis of prescription fill times is based on 
one data set that follows each individual prescription in calendar year 2014 (CY2014) 
through the entire prescription fill process. Because data from prior to the implementation 
of automation is unavailable, medication demand and prescription characteristics are 
estimated from this post-automation data to simulate the pre-automation process. 

On average, prescriptions filled automatically in CY2014 are completed 
6.97 ±0.97 (standard error) minutes faster than prescriptions filled manually in CY2014. 
During periods of high demand, automatic fills were quicker than manual fills 
84.5 percent of the time, with a 95 percent confidence interval of [80.2, 87.4]. On 
average, prescriptions filled after the installation of automation are 4.4 ±0.43 (standard 
error) minutes faster than prescriptions filled prior to the installation of automation. 
These results are found to be statistically significant. 

The difference in daily average prescription fill times pre-automation and daily 
average prescription fill times post automation is used as the response in a linear 
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regression to determine which characteristics of the data most heavily influence this 
difference; a negative value of the response represents a day in which the average 
prescription fill time pre-automation is quicker than post-automation. Results indicate the 
number of hours of high demand experienced in the day negatively impacts the response, 
while the post-automation proportion of medications filled through automation and the 
pre-automation proportion of highly prescribed medications positively impact the 
response. 

With all other factors held constant, an automation workload proportion of 0.37 is 
related to an increase in the difference of average prescription fill time of 2.98+ 1.95 
(standard error) minutes; if this proportion can be maintained over the course of an entire 
year, assuming an average daily weekday demand is 1186 prescriptions, this alone could 
reduce yearly workload by 636 days, compared to pre-automation. Assuming there are 
25 pharmacy technicians working within the pharmacy, this is also associated with a 
reduction in daily workload of 2.34 + 0.03 hours per technician. Figure 1 illustrates the 
yearly reduction in weekday workload and daily reduction in workload based on 
proportion of prescriptions filled through automation. 



Figure 1. Estimated Yearly and Daily Reduction (per Technician) in Weekday 
Workload Based on Proportion of Prescriptions Filled through 
Automation Demand. 
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It is also important to consider the implementation from a cost perspective. The 
pharmacy automation system at NMSCD carried an individual implementation cost of 
$2.4 million of the total $49 million contract. There is a reduction in total time spent 
filling prescriptions of 14.2 percent from pre-automation to post-automation. This 
reduction in time is associated with $330,000 reduction in cost, the amount of money it 
would cost to pay for the equivalent amount of work from enlisted military pharmacy 
technicians. With these calculations, it would take roughly seven years to recoup the cost 
of implementation at NMCSD based on increased efficiency. This cost analysis does not 
account for efficiencies based on the storage and supply of medications, increased patient 
satisfaction, increased access to care, increased accuracy of dispensed medications, or 
recapture costs based on a reduction of non-military prescription fills. 

After the implementation of pharmacy automation at NMCSD, there is an 
increase in efficiency, defined as the total prescription fill time. This efficiency can be 
quantified as reduction in total time to fill a prescription, FTEs, or equivalent salary of 
pharmacy technicians. This data is collected after a little more than one year of 
implementation; with better training and more emphasis placed on the optimal 
automation workload, these savings in efficiency will continue to increase. 
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I. 


INTRODUCTION 


In today’s healthcare environment, healthcare facilities continue to seek ways to 
reduce cost and improve efficiency; the pharmacy is one of many avenues to accomplish 
this task. The current pill-counting machines in most pharmacies within Navy Medicine 
are obsolete, and more advanced replacement technology is available. In particular, 
facilities that service a larger population of customers have a greater opportunity to 
benefit from technological advances in pharmacy automation, because their pharmacies 
experience a higher demand. Replacing legacy systems with updated technology could 
have extreme impacts on efficiency, accuracy, patient satisfaction, and access to care. 

Prior to 2012, pharmacy technicians manually filled all prescriptions at the Naval 
Medical Center San Diego (NMCSD) pharmacy, supplemented by an AccuMed pill¬ 
counting machine. This pharmacy received an upgrade in August 2012, when the Bureau 
of Medicine and Surgery (BUMED) installed a state of the art pharmacy automation 
system at this location. The new automation machine is capable of autonomously 
completing all steps of the prescription fill process prior to pharmacist verification 
without assistance from pharmacy technicians. Because every prescription filled using 
this new technology collects data through barcode scanning, its implementation provides 
a unique opportunity to objectively evaluate its contributions to efficiency. 

A. OBJECTIVES AND METHODOLOGY 

The objective of this study is to quantify the increases in efficiency after the 
installation of pharmacy automation at NMCSD. In order to evaluate these objectives, 
this study focuses on individual prescription orders filled at NMCSD during calendar 
year 2014 (CY2104), from January to December 2014. Because this data set only 
contains information for one year post-automation, a pre-automation data set is generated 
through simulation for use as comparison. All statistical analysis and simulation is 
completed using R (2013). 

A customer placing a prescription order at the intake window in the pharmacy 
signals the beginning of the prescription fill process; the total prescription fill time is of 
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interest, which is the difference in time between the placement of an order and the 
completion of pharmacist verification. To evaluate the efficiency of the machine, the total 
prescription fill times for all observed manual prescriptions fills in CY2014 are compared 
to the total prescription fill times for all observed automatic prescriptions fills in CY2014. 
Similarly, the entire post-automation data set is also compared to the simulated pre¬ 
automation process. The results of these comparisons determine if the differences are 
statistically significant. 

B. SCOPE, LIMITATIONS, AND ASSUMPTIONS 

This study focuses solely on data obtained from NMCSD. Because this study 
focuses on one specific location, its results can only be directly applied to NMSCD; 
however, medical facilities of similar size and patient population should expect to see 
similar increases in efficiency after the installation of a pharmacy automation system at 
their location. The techniques used to evaluate the efficiency of the automation system at 
NMCSD can also be easily applied to another location with a similar automation system. 

Data from prior to the installation of the automation system is unavailable, 
leading to the most significant limitation of this study: if this data was obtainable, a 
considerably better comparison of changes in efficiency could be completed. In order to 
address this limitation, manual fill times and daily demand distributions are estimated 
from the existing data and used to construct a simulation of the pre-automation process. 
In addition, with only one complete year of post-automation data, it is impossible to 
validate significant departures from the mean average fill time observed during specific 
months as a recurring trend. To compensate for this volatility, the month of February is 
eliminated from the data set for most of the analysis to prevent this month with unusually 
high prescription fill times from unduly influencing the results. 

The most basic assumption of this study is that the automation machine is running 
properly and efficiently for the entire data collection period. In the absence of 
information on the breakdown and repair of the machine, this assumption is necessary to 
conduct analysis. In addition, the use of simulation to generate information for the pre¬ 
automation process requires two major assumptions. First, it is essential to assume that 
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there is no significant variability in the prescription filling abilities of the pharmacy 
technicians performing manual prescription fills in CY2014. Second, because the demand 
in the simulation is estimated from the demand observed in CY2014, it is assumed that 
demand experienced during that year is characteristic of any year in the NMCSD 
pharmacy. Additional assumptions required by this study are addressed as necessary. 

C. COURSE OF STUDY 

This study consists of four additional chapters. Chapter II, Background, explains 
the history of the implementation of the pharmacy automation system at NMCSD and 
presents a review of existing literature on the topic of pharmacy automation. The 
objective of Chapter III, Data, and Chapter IV, Methodology and Analysis, is to quantify 
the efficiencies experienced after the implementation of automation. Chapter V, Results, 
Conclusions, and Recommendations, highlights the results and conclusions of the study, 
makes recommendations about automation within Navy Medicine, and suggests topics 
for future work. 


3 



THIS PAGE INTENTIONALLY LEET BLANK 


4 



II. BACKGROUND 


All pharmacies face a variety of concerns, such as quality assurance, pharmacy 
workflow, patient satisfaction, and cost control. With these challenges in mind, the 
Bureau of Medicine and Surgery (BUMED) looked toward pharmacy automation to 
upgrade their prescription fulfillment process at several high-volume sites throughout the 
enterprise. Specifically at NMCSD, this upgrade converted the pharmacy from a manual 
system to an automated system, supplemented by pharmacy technicians. 

A. MANUAL PRESCRIPTION FILLS 

Prior to 2012, pharmacy technicians manually filled all prescription orders within 
the NMCSD pharmacy. To fill a prescription manually, pharmacy technicians received 
the order, obtained the specific medication from its place of storage, and then physically 
counted each pill; thirty-six AccuMed automated pill counting machines supplemented 
the pharmacy technicians in NMCSD, which simply counted a specific number of pills, 
loaded by the user (St. Onge Company, 2011). Once counting was completed, the 
technician bottled the medication, capped the bottle, and then printed and affixed the 
label. At this stage, all bottled prescriptions were physically carried to the pharmacist for 
verification, who certified the medication filled fit the prescription entered for each 
customer. Manual completion required the pharmacy technicians and pharmacists to 
physically complete each step in the process, as well as retrieve medications from the 
storage shelves and walk the medication between the different stages. 

The customer experienced a “bank-teller” queuing system within the NMCSD 
pharmacy while getting a prescription filled. Specifically, when a customer first arrived 
in the pharmacy, he entered a queue to be received at the window. Once called to the 
window, he placed his order, and then waited at the window while the order was filled. 
When the order was completed, the customer’s prescription was reviewed with him, he 
accepted the order, and then he departed the pharmacy (B. Detlef, BUMED, personal 
communication, July 30, 2015). Eigure 1 illustrates both the customer queuing system 
and the manual fill process. 


5 




Figure 1. Bank-Teller Queuing System with Manual Fill Process. Once a 

customer places an order, he waits at the dispensing window while 
his prescription is filled manually. 


B. AUTOMATION IMPLEMENTATION 

In August 2012, the United States Navy awarded a $49 mi llion contract to 
Innovation, makers of the PharmASSIST pharmacy automation technology, to supply 
pharmacy automation to its high-volume sites (Innovation, 2012). This included the 
installation of pharmacy automation in NMCSD in 2013; this site currently has the most 
technologically advanced and fully functional system in Navy Medicine. This automation 
system is capable of handling and storing 360 unique medications, and consists of a 
conveyer system that is activated once a customer places an order (B. Detlef, BUMED, 
personal communication, April 8, 2015). The conveyor system is not used solely for 
prescription fills that have been automatically counted, capped, and labeled; pharmacy 
technicians also use the conveyer system to quickly route manual prescription fills. 
Figure 2 is a photograph of the actual conveyer system installed at NMCSD. 
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Figure 2. Conveyer Belt Installed at Naval Medical Center San Diego. Once 
the customer places an order, the conveyor system directs each 
prescription through the different steps of the fill process. Manual fill 
stations are located to the right side of the conveyer belt; as 
pharmacy technicians complete a manual fill, they place completed 
orders onto the conveyer belt for routing to pharmacist verification. 


Once the customer order is loaded, the automation system dispenses a vial, affixes 
the label, and delivers the empty bottle to the Robotic Delivery System (RDS). The RDS 
inserts the labeled vial into the inventory container containing the appropriate medication, 
dispenses the pills, caps the vial, releases the vial, and routes the medication to the 
pharmacist for verification. Figure 3 shows the RDS inserting the labeled bottle into an 
inventory container. This automatic procedure is used to supplement the manual fill 
process described in Section A. 
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Figure 3. Robotic Delivery System Installed at Naval Medical Center San 

Diego. Each container houses an inventory of medication. The RDS 
selects a vial, inserts it into the container to receive dispensed 
medication, caps the vial, and affixes the label. This completed 
prescription is then placed on the conveyer belt for routing to 
pharmacist verification. 


Along with the implementation of the pharmacy automation system, NMCSD 
underwent changes to their customer queuing system; instead of a bank-teller system, 
customers experience an “in-and-out” system. When a customer first arrives in the 
pharmacy, he takes a ticket number from an automated kiosk, and waits for that ticket 
number to be called. Once the number is called, the customer is received at the window, 
where he places his order. This signals the beginning of the prescription fill process. The 
customer is dismissed from the window, where he waits for his ticket number to be called 
for a second time. After his ticket number is called again, the customer enters a queue to 
wait for order pickup. At his turn, the customer is received at the dispensing window, the 
customer’s prescription is reviewed with him, he accepts the order, and then he departs 
the pharmacy (B. Detlef, BUMED, personal communication, July 30, 2015). Eigure 4 
illustrates the customer queuing process combined with the manual and automatic fill 
process. By dismissing a customer from the window, this queuing system allows for 
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greater window utilization than the bank-teller model; the customer is not occupying a 
window while prescription operations are occurring. 



Figure 4. In-and-Out Queuing System with Manual and Automatic Fill 

Process. Using the in-and-out model, the queuing process for the 
customer is different, and the prescription fill process is more 
streamlined. The automation is capable of performing all steps, 
including transportation, up to pharmacist verification via the 
conveyer belt. Manual fills are located along the conveyor system 
for increased efficiency. 


While the automation system at NMCSD does not completely eliminate the need 
for manual prescription fills, it has the potential to greatly reduce the workload placed on 
pharmacy technicians. This reduction in workload occurs not only in the physical number 
of orders filled manually, but also decreases the need for pharmacy technicians to 
constantly retrieve and stock medication; most of the high-demand medications are stored 
within the RDS system. In addition, each step in the automatic process is triggered by 
barcode scanning, which, in theory, should lead to more accurate prescription fills. 
Through barcode scanning, the PharmASSIST software ensures that every step in the 
process is automatically double checked for accuracy prior to the pharmacist verification. 
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c. 


LITERATURE REVIEW 


There exists extensive literature surrounding the implications of changing 
pharmacy operations within a hospital; much of this literature specifically identifies the 
outcomes related to the implementation of different automation systems. Due to an 
environment of rapidly changing technology, the ability to understand the impact of such 
a change is of significant interest to any healthcare organization. A review of existing 
literature concerns the topic of efficiency. This review is limited to studies conducted in 
hospital pharmacies as opposed to community pharmacies, which are of similar size and 
scope to NMCSD, but are varied in the types of automation systems implemented. 

The use of emerging technologies within pharmacies is, in part, directly aimed at 
increasing productivity while reducing customer wait times. One way to quantify the 
productivity of the pharmacy is to measure the total output of prescription items over 
time; through statistical inference, the quantity of items produced prior to installation can 
be compared to after installation and evaluated for significance. Though many other 
factors, such as staff training and competency, can impact this metric, the impact of 
automation on the number of items produced per day varies widely from study to study. 
This increase can range from as little as nineteen percent to as much as 43 percent; see 
Fitzpatrick, Cooke, Southall, Kauldher, and Waters (2005), Angelo, Christensen, and 
Ferrerri (2005), and James et al. (2013). Many of these numbers have significant standard 
deviations, and items produced per hour can vary by as much as 68 percent of the mean 
(Angelo, Christensen, & Ferreri, 2005). In addition, staffing practices must be adjusted to 
maximize this effect on productivity. While automation can reduce the average time 
spent filling prescriptions by seventeen percent after the introduction of automation, this 
reduction is associated with eighteen percent increase in time spent doing non-productive 
activities if staffing levels remain unchanged (Lin, Huang, Punches, & Chen, 2007). 

Because of the cost associated with automation machines, many healthcare 
organizations turn to simulation as a reduced cost method to influence decision-making. 
With proper modeling and input parameters, simulations can provide valuable insights 
into important aspects of pharmacy management, especially when used as a queuing 

model. Customer satisfaction surveys indicate that pharmacy customers value a wait time 
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of less than 30 minutes as the third most important factor in a pharmacy experience, after 
only prescription accuracy and affordability of medicine (Vincent & Lim, 1997). This 
goal of a 30-minute customer wait time is used as target metric in Tan, Chua, Yong, and 
Wu’s simulation of the impacts of automation on a hospital pharmacy in Singapore 
(2009). The results of this simulation indicate that the implementation of automation 
alone is not substantial enough to reduce the 95* percentile customer wait time to below 
30 minutes; in addition to automation, pharmacies must change their staffing procedures 
and workflow management to achieve this goal (2009). 

Based on the unavailability of data collected prior to the installation of the 
automation machine at NMCSD, this study takes a hybrid approach, incorporating both 
empirical evaluation and simulation. The post-automation data indicates whether a 
prescription is filled manually or automatically; therefore, inferences are made about 
manual fill times, in general. The data also provide demand signals that assist in 
determining customer requests on an hourly, daily, and weekly basis. This information is 
used simulate the pre-automation environment, which is the basis for statistical inference 
in Chapter IV. 

D. SUMMARY 

The efficiency of medication dispensed in a hospital pharmacy will always be of 
the utmost importance to any healthcare organization. Navy Medicine has attempted to 
address these issues through the implementation of pharmacy automation of varying 
degrees at its high volume sites. At NMCSD, in particular, the most advanced and fully 
functional automation machine has been in use for nearly three years; this type of 
automation has the potential to greatly reduce workload on pharmacy technicians and 
pharmacists, which in turn, will increase efficiency and patient satisfaction. These 
benefits will be addressed in the following chapters. 
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III. DATA 


Analysis of prescription fill times is based on one data set generated by 
Innovation, the system that runs the automation machine procedures to fill, verify, and 
stage prescriptions at NMCSD. Improvement Path Systems, an organization contracted to 
perform healthcare analytics, provided this data via the BUMED. This data set follows 
each individual prescription in CY2014 through the entire fill process; the time lapse 
between automation implementation in 2012 and data collection in 2014 reduces the 
possibility of an adjustment period influencing the results. Specifically, this accounts for 
technician training and modifications to standard operating procedures. Medication 
demand and prescription characteristics are estimated from this post-automation data to 
simulate the pre-automation process, because data from prior to the implementation of an 
automation system is unavailable. Using both the simulated pre-automation and observed 
post-automation data, it is then possible to conduct the statistical analysis completed in 
Chapters IV and V. 

Each observation, 403,900 in total, represents an individual prescription request at 
the pharmacy during CY2014, and includes a unique prescription identification number, 
dispensed medication name, and fill type (manual or automatic). In addition to these 
descriptive fields, the data set contains fifteen tracked processes for each prescription. 
Each process is represented by the date and time in which the process occurred, accurate 
to the second. Table 1 details the eight processes relevant to this study; other processes of 
interest, but not used to this study, are found in Appendix A. 
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Table 1. Processes Tracked in Prescription Data. The prescription data 

contains the date and time, accurate to the second, of eight relevant 
processes during the entire prescription order procedure. 


Process 

Description (date/time) 

Scheduled 

Entered into the system 

Counted 

Count of quantity of medication is complete 

Added to Tote 

Counting, bottling, capping, and labeling is complete 

Verified 

Pharmacist verification for errors is complete 

Added to Bag 

Combined with remaining customer order 

Added to Location 

Complete order is placed in assigned cubby location 

Ready for Pickup 

Order is complete and ready for customer pick up 

Pickup 

Customer picks up completed order. 


Figure 5 illustrates the flow of the prescription order process within the pharmacy. 
For this analysis, total time to fill a single prescription is of interest. Total time is defined 
as the difference in time between scheduling and verification; all processes after that are 
possibly dependent on the other items contained in a customer order, or the actions of the 
customers themselves. The difference in scheduled time and verified time is found for 
each observation, and this time difference is used as the response during analysis 



Figure 5. Prescription Fill Flow Chart. The general prescription fill process is 
outlined, including indication of which steps in are recorded in the 
data set. 
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Of the 403,000 observations identified in the post-automation preseription data, 
32,841 items are eliminated based on issues with observed scheduled, verified, and 
pickup times. Observations without these times recorded are excluded. In addition, all 
observations with another process recorded at a point in time prior to scheduling are 
removed; scheduling is the very first step in the prescription fill process, and observations 
contrary to this are treated as errors. These times are important because their difference is 
used as the measure of total prescription fill time. Another 109 observations are 
eliminated because the type of fill, either manual or automatic, is not specified. Lastly, 
7,017 observations have a total prescription fill time of either less than two minutes or 
greater than one day. Because the median prescription fill time is 16.3 minutes, including 
these observations, observations greater than one day are considered extreme outliers and 
excluded from analysis. Similarly, based on the number of steps in the prescription fill 
process, any observation of less than two minutes is uncharacteristically quick. The result 
is a final data set consisting of 364,529 observations, as illustrated in Figure 6. 



Figure 6. Reduction of403,900 Initial Observations to Final Data Set of 

364,529. An analysis of available prescription data results in a final 
sample size of 364,529 observations for CY2014 at NMCSD. 
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In the final data set, the automation machine completes 37.3 percent of all 
prescription fills; the rest are filled manually. Summary statistics for the final data set, 
distinguishing prescriptions filled automatically from manually, are contained in Table 2. 


Table 2. Summary Statistics of Prescription Fill Times (Minutes). In the 

final data set, 37.3 percent of all prescription fills for CY2014 are 
filled through automation. 


Type 

Number 
of Fills 

Minimum 

1st 

Quartile 

Median 

Mean 

3rd 

Quartile 

Maximum 

Total 

364,529 


8.95 

15.98 

26.09 

28.10 

1440.00 

Auto 

136,068 

2.00 

7.07 

12.90 

21.74 

23.48 

1435.00 

Manual 

228,461 


10.38 

17.95 

28.68 

30.75 

1440.00 


A. POST-AUTOMATION PRESCRIPTION FILL 

The final data set contains the entirety of information used to analyze post¬ 
automation prescription fills. It is of interest to examine trends in demand and 
prescription fill times based on different time periods; assuming these tendencies are 
consistent with the typical pharmacy environment at NMCSD, they are used as the basis 
to simulate the pre-automation process. Data is binned by day, day of the week, and hour 
of the day to determine significant time periods in order to better understand the 
workload and demand signals within the pharmacy. 

I. Daily Trend Analysis 

First, the number of prescriptions observed is totaled by day of the year. This 
procedure results in 365 observations, and serves as the basis for the first stage of 
analysis, see Figure 7. Upon inspection, a weekly trend is apparent; there are roughly four 
local minima and maxima during each month, corresponding to the number of weeks in a 
month. In addition, the number of prescriptions appears to increase in the month of 
February, while the rest of the year remains fairly consistent. 
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Figure 7. Total Number of Prescriptions. This includes prescriptions filled 
both manually and automatically. The maximum number of daily 
prescriptions is 1,911 observations, occurring on February 12, 2014. 
The minimum number of prescriptions is 246, occurring on 
November 27, 2014. The sharp increases and decreases in the 
number observations during each month indicate a weekly trend. 


Similarly, prescription fill time is averaged by day of the year, resulting in 
365 observations; see Figure 8. Average daily prescription fill times experience an 
increase during the month of February, while they are at their lowest during the month of 
August. This spike in prescription fill time during the month of February corresponds 
with an increase in customer orders during that month. With only one year of 
observations, this cannot be validated as a recurring seasonal trend. This departure from 
the mean proves to be problematic throughout this study; as a result, the month of 
February is eliminated during most analysis conducted in Chapters IV and V. 
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Figure 8. Average Prescription Fill Times. This includes prescriptions filled 
both manually and automatically. The maximum average fill time of 
57.9 minutes occurred on February 5, 2014, while the minimum of 
9.04 minutes occurred on August 10, 2014. 

2. Day of Week Trend Analysis 

Next, the number of prescriptions filled and their fill times are analyzed based on 
day of the week. It is noted previously that there is a significant weekly trend in the 
number of prescriptions filled within a day. The pharmacy at NMCSD is open seven days 
a week; however the regular workweek for the base is Monday through Friday. 
Therefore, it is expected that there are significantly fewer customer orders on weekends 
than on weekdays, because patients tend to make appointments during the business week. 
In order to validate this expectation, observations are binned by day of the week, and then 
averaged within that bin. Demand decreases on the weekends, which is also associated 
with a lower average prescription fill time. Of more interest, the average prescription fill 
time on Mondays is higher than other weekdays without a corresponding increase in 
demand. Figure 9 illustrates the effect that day of the week has on average number of 
prescriptions filled and average prescription fill time. 
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Figure 9. Average Prescription Fill Time and Average Number of Customer 
Prescription Orders, Binned by Day of Week. The black bars 
represent standard error of the mean. Demand decreases on 
weekends, which is also associated with a lower average prescription 
fill time. On weekdays, demand for prescriptions remains fairly 
constant. Mondays have an increased average prescription fill time 
without a corresponding increase in demand. 


3. Hour of Day Trend Analysis 

A normal workday for the outpatient pharmacy at NMCSD begins at 0800 and 
ends at 2100, Monday through Friday. On Saturday and Sunday, business hours are from 
0800 to 1800. At all other times, the outpatient pharmacy is officially only open for 
prescription fills associated with Emergency Department visits. Because of this schedule, 
it is expected that customer demand will be higher during working hours and lower 
during non-working hours. In order to validate this expectation, observations are binned 
by hour of the day, and then averaged within that bin. Prescription demand experiences a 
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sharp increase at 0800, and then tapers off after 1700. Prescription fill times experience 
delays around 0700 and 1800. Figure 10 illustrates the relationship between hour of the 
day, prescription demand, and prescription fill times. 




Figure 10. Average Prescription Fill Time and Average Number of Customer 
Prescription Orders, Binned by Hour of the Day. The black bars 
represent standard error of the mean. Weekends and weekends are 
separated because there are different working hours, depending on 
the day. Prescription demand experiences a sharp increase at 0800, 
and then tapers off after 1700. Prescription fill times experience 
delays around 0700 and 1800. 


Based on the observed increase in preseription fill times without an increase in 
demand on Mondays, this day is evaluated individually. Figure 11 illustrates the average 
prescription fill time, per hour, during working hours on Monday; there is an extreme 
increase in the fill time from 1700 to 1900. This hourly increase directly contributes to 
the overall increase in average daily prescription fill times on Mondays, compared to the 
rest of the week. 
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Figure 11. Average Hourly Prescription Fill Time During Working Hours, 

Monday. The black bars represent standard error of the mean. There 
is an extreme increase in the average hourly prescription fill time 
from 1700 to 1800, where the fill times jumps to a maximum of 
147 minutes. 


B. PRE-AUTOMATION PRESCRIPTION FILL 

A significant limitation of this study is the unavailability of data collected pre¬ 
automation. To evaluate the effectiveness of the automation system, a pre-automation 
data set is constructed using simulation to model the process prior to automation 
implementation. Two major assumptions are made in order to construct this simulation: 
first, the CY2014 medication demand is representative of typical yearly demand, and 
second, the manual fill times of the pharmacy technicians in CY2014 is characteristic of 
typical manual fill times per prescription. 

I. Design 

Using the name of the dispensed medication as an identifier in the CY2014 data 
set, it is observed that there are 2,340 unique medications—the same type of medication 
in a different dosage is counted as a unique medication. Of those medications, the ten 
most prescribed medications account for 21.3 percent of the total prescription in CY2014; 
a distribution of the 25 most prescribed medications can be found in Appendix B. 
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Prescription fill time is dependent on the type of medication ordered. Figure 12 and 
Figure 13 illustrate the difference in manual prescription fill time and demand for four of 
the most frequently filled medications. 



Figure 12. Histogram of Daily Average Manual Fill Time (Minutes) for Four 
Frequently Prescribed Medications. The manual fill time varies 
based on medication. 
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Figure 13. Prescription Demand for Three Frequently Prescribed Medications. 

Demand for different medications varies throughout the year, based 
on the unique medication. For example, the demand for 
Hydrocodone/Acetaminophen 326MG tends to be larger than the 
demand for Acetaminophen 325MG during the middle portion of the 
year. 


Monthly and weekly trends are used as the basis for simulation; it is not essential 
to detail hourly trends at this level. Because it is assumed that the demand and manual 
prescription fill times are consistent with CY2014 data, it is only necessary to simulate 
the extra demand placed on the pharmacy technicians in the absence of automation. For 
this simulation, this means that the actual fill times observed in CY2014 for manual 
transactions each day of the year are left unchanged; only the additional manual workload 
not supplemented by the automation machine is simulated. The variables of interest for 
the simulation include total prescription demand, medication demand, and prescription 
fill times, as defined in Table 3. 


23 











Table 3. Indices, Derived Data, and Calculated Variables for Simulated 
Pre-Automation Process. The additional manual workload not 
supplemented by the automation machine is simulated to create a 
pre-automation data set. 



Notation 

Description 

Indices 

i 

Day of the Year, i g{1,...,365} 

j 

Day of the Week, j g{ 1,...,7} 
where j = 1 is Sunday 

k 

Month, k g {1,...,12} 

m 

Medication, m g {1,...,2340} 

Derived 

Data 

total _ fills ^ 

Total number of prescriptions 
filled on each day of the year, i. 

manual _ fills- 

Total number of prescriptions 
filled manually on each day of the 
year, i. 

medication_demand ^ 

Vector of medications demanded 
on each day of the week, j, for 
each month of the year, k. 

manual_lill_time^ . ^ 

Vector of manual fill times 
associated with medication m for 
each day of the week, j, for each 
month of the year, k. 

Calculated 

Variables 


Standard deviation of daily 
demand on each day of the week, j, 
and month, k. 

upper _boundj 

Upper bound for simulated total 
prescription demand on each day 
of the year, i. 

lower _bound- 

Lower bound for simulated total 
prescription demand on each day 
of the year, i. 

simulated _total^ 

Total simulated prescription 
demand on each day of the year, i. 

excess _ manual- 

Excess manual workload on each 
day of the year, i. 
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First, the medication demand at NMCSD pharmacy in CY2014 is aggregated 
by day of the week, j, and month, k, resulting in 84 vectors of varying length, 
medication_demand^. ^. This vector contains the names of all medications filled on day 

of the week, j, in month, k; individual medications appear in this vector with the same 
frequency as observed in CY2014. Similarly, all manual fill times associated with the 
demanded medications are aggregated by medication, m, day of the week, and month of 
the year, This results in 196,560 vectors of varying length, one for 

every medication, m, on every day of the week, j, in each month, k. Then, total number of 
prescription fills and manual prescription fills completed on each day of the year, i, are 
calculated. This results in 365 observations of two variables, total_ fills^ and 
manual_ fills respectively; on average, the automation machine accounts for 
37.3 percent of prescription fills per day. The total number of prescriptions ordered on 
a specific day of the week within each month is modeled as uniformly distributed. The 
lower and upper bounds of the uniform distribution are calculated the observed total fill 
for that day, total_ fills-, and the standard deviation of the daily demand corresponding to 
the day of the week and month, sdj ^; this is shown in Equations (1) and (2). 

upper _ bounds - total _ fillsi + sdj f, (1) 

lower_bound. - total_ fills^ - sdj ^ (2) 

jG{l,...,365},yG{l,...,7},kG{l,...12} 

For each day of the year, a random uniform number within the calculated lower 

and upper bounds is drawn to determine the day’s simulated total demand, 

simulated _total-. Because the observed manual prescription fills from CY2014 will 
remain unchanged, the observed manual fills, manual _ fills, are removed from the 
simulated total demand, leaving only the excess workload per day, excess_manual., as 
shown in Equation (3). 

excess_manual- = simulated_total- - manual_ fills-, i g {1,..., 365} 
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(3) 



For each day of the year, a random medication sample of size excess_manual- is 
selected at random from medication_ demand^ based on the appropriate day of the 

week and month. Using the medications in the sample, manual fill times are chosen with 
replacement from manual _ fill _ time the actual manual fill times for each 

medication observed during that month and day of the week. If a medication does not 
have an associated manual fill time for a specified day of the week within month k, the 
manual fill time is assigned as the average manual fill time for the specified medication 
in month k. If a medication does not have an associated manual fill time in all of month k, 
the manual fill time is assigned as the average manual fill time for the specified 
medication over the entire year. If a medication does not have an associated manual fill 
time in the entire year, the manual fill time is assigned as the average manual fill time for 
all medications; there are only seventeen medications in CY2014 filled solely through 
automation, accounting for 0.07 percent of total observations. This entire process is 
repeated with 1,000 replications. 

Through this technique, 135,972 data points are created using simulation, 
representing the excess demand within the pharmacy not supplemented by automation. 
These simulated observations are then combined with the observed manual prescription 
fill times, resulting in a pre-automation data set consisting of 364,443 total data points. 
Figure 14 illustrates the observed and simulated prescription demand. 
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Figure 14. Simulated Excess Demand Combined with Observed CY2014 

Prescription Demand. Through simulation, the excess demand not 
supplemented by automation is generated to produce a pre¬ 
automation data set. 

2. Validation 

In order to confirm the validity of this simulated process, the automation demand 
for CY2014 is generated through a similar technique, using only monthly and weekly 
trends. Again, the manual fill times observed in CY2014 are left unchanged; only the 
automatically filled prescriptions are simulated. By simulating the automated prescription 
fills and combining these data points with the observed manual fill times, simulated data 
set can be compared to the observed CY2014 data. The variables of interest for this 
validation include total prescription demand, medication demand, and prescription fill 
times, as defined in Table 4. 
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Table 4. Indices, Derived Data, and Calculated Variables for Simulated 
Automation Process. The additional manual workload not 
supplemented by the automation machine is simulated to create 
automation data. 



Notation 

Description 

Indices 

i 

Day of the Year, i g{1,...,365} 

j 

Day of the Week, j g{1,...,7} 

where j= 1 is Sunday 


k 

Month, k g {1,...,12} 


m 

Medication, m g {1,...,2340} 


total _ fills ^ 

Total number of prescriptions 
filled on each day of the year, i. 

Derived 

Data 

manual _ fills- 

Total number of prescriptions 
filled manually on each day of the 
year, i. 

medication_demand ^ 

Vector of medications demanded 
on each day of the week, j, for 
each month of the year, k. 


auto_fdl_time^ ^ ^ 

Vector of automatic fill times 
associated with medication m for 
each day of the week, j, for each 
month of the year, k. 



Standard deviation of daily 
demand on each day of the week, j, 
and month, k. 


upper _bound- 

Upper bound for simulated total 
prescription demand on each day 
of the year, i. 

Oitlculct 

Variables 

lower _bound- 

Lower bound for simulated total 
prescription demand on each day 
of the year, i. 


simulated _total^ 

Total simulated prescription 
demand on each day of the year, i. 


simulated _auto- 

Simulated automatic workload on 
each day of the year, i. 


The medication_demand^. ^ vector remains unchanged for validation. All 

automatic fill times associated with the demanded medications are binned by medication, 
day of the week, and month of the year, auto_filI_time^ ^ ^. This results in 196,560 

vectors of varying length, one for every medication, m, on every day of the week, j, in 
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each month, k. Again, the total number of prescriptions ordered on a specific day of the 
week within each month is modeled as uniformly distributed. The lower and upper 
bounds of the uniform distribution are calculated the observed total fill for that day, 
total_ fills., and the standard deviation of the daily demand corresponding to the day of 
the week and month, sdj ^; this is shown in Equations (1) and (2). 

For each day of the year, a random uniform number within the calculated lower and 
upper bounds is drawn to determine the day’s simulated total demand, simulated_total.. 
Because the observed manual prescription fills from CY2014 will remain unchanged, the 
observed manual fills are removed from the simulated total demand, leaving only the 
automation workload, simulated_auto^, as shown in Equation (4). 

simulated_auto^ = simulated_total. - manual_fidls., i e {1,...,365} (4) 

For each day of the year, a random medication sample of size simulated_auto. is 
selected at random from medication_ demand^ based on the appropriate day of the 

week and month. Using the medications in the sample, automatic fill times are chosen 
with replacement from auto_fill_time^^^, the actual automatic fill times for each 

medication observed during that month and day of the week. If a medication does not 
have an associated automatic fill time for a specified day of the week within month k, the 
fill time is assigned as the average automatic fill time for the specified medication in 
month k. If a medication does not have an associated automatic fill time in all of month k, 
the fill time is assigned as the average automatic fill time for the specified medication 
over the entire year. If a medication does not have an associated automatic fill time in the 
entire year, the fill time is assigned as the average automatic fill time for all medications. 
This entire process is repeated with 1,000 replications. 

Through this technique, 136,074 data points are constructed using simulation, 
representing the automation workload within the pharmacy. These simulated observations 
are then combined with the observed manual prescription fill times, resulting in a data set 
consisting of 364,535 total data points. Figure 15 illustrates the observed daily demand 
and the daily demand with simulated automation workload; Figure 16 illustrates the 
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observed average daily prescription fill time and the average daily prescription fill time 
with simulated automation workload. 



Jan l-cb Mar Apr .May Jun Jul Aug Sep Oct Nov Dec 

Figure 15. Observed and Simulated Daily Demand. With the same simulation 
techniques used to create the pre-automation manual workload, a 
simulated automation workload is generated. 



Figure 16. Observed and Simulated Automation Average Prescription Fill 

Time. With the same simulation techniques used to create the pre¬ 
automation manual workload, a simulated automation workload is 
generated. 
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This simulated automated workload provides a degree of validity to the 
techniques used to simulate the pre-automation data. The simulated automatic data set 
tends to overestimate both the daily demand and the average daily prescription fill time; 
this suggests that the simulated pre-automation data set will be a upper bound of the 
manual workload. The absolute mean difference between daily observed demand and 
daily simulated demand is 72.9 + 51.2 (standard deviation) items; 70 percent of the 
differences fall within 100 items. February is a month with uncharacteristically large 
differences, and if excluded from the data set, the mean difference between daily 
observed demand and daily simulated demand is 67.4 + 40.6 items. Figure 17 illustrates 
the difference in the demand for the observed and simulated data sets throughout an 
entire year. The mean difference between average daily observed prescription fill time 
and average daily simulated prescription fill time is 6.3+4.8 minutes; 83 percent of 
observations fall within ten minutes. With February excluded, the mean difference 
between average daily observed prescription fill time and average daily simulated 
prescription fill time is 5.4+ 2.9 minutes. Figure 18 illustrates the difference in 
prescription fill time for the observed and simulated data set throughout an entire year. 
Evaluation of the difference in the observed and simulated data set aggregated by month 
and day of week, is conducted; other than the month of February, there does not appear to 
be any patterns in the average differences based on month or day of week. These plots 
can be found in Appendix C. 
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Figure 17. Difference in Daily Observed Demand and Daily Simulated 
Demand. After simulating the automated workload, the mean 
difference between the daily observed demand and the daily 
simulated demand 72.9 observations with a standard deviation of 
51.2 observations. The month of February is extremely volatile; 
without this month, the mean difference is 67.4 observations with a 
standard deviation of 40.6 minutes. 


67.4 
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Figure 18. Difference in Average Daily Observed Prescription Fill Time and 
Average Daily Simulated Prescription Fill Time. After simulating 
the automation workload, the mean difference between the observed 
average daily prescription fill time and the simulated daily 
prescription fill time is 6.3 minutes with a standard deviation of 4.8 
minutes. The simulated data set is more volatile during the month of 
February; without this month, the mean difference is 5.4 minutes 
with a standard deviation of 2.9 minutes. 


To determine if there are any systematic trends over time, two statistical tests are 
performed on the difference in average daily observed prescription fill time and average 
daily simulated prescription fill time; February is an extremely volatile month, and is 
removed from the data set. First, a two-sided runs test for detecting non-randomness is 
performed, and the results indicate there is no evidence to suggest dependence in the 
differences over time (Bradley, 1968). The standardized runs test statistic is -1.36 with a 
p-value of 0.17. Next, a Durbin-Watson test is conducted to detect autocorrelation, and 
the results are consistent with the two-sided runs test, indicating independence in the 
differences over time (Durbin & Watson, 1950). The Durbin-Watson test statistic is 
1.98 with a p-value of 0.44. The difference in average daily prescription fill time and 
average daily simulated prescription fill time does not show any evidence of non¬ 
randomness or autocorrelation. 
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C. SUMMARY 

There are significant trends in the demand and prescription fill times of 
prescriptions filled at NMCSD in CY2014. With these trends, monthly and weekly 
indicators are used to generate the increased manual demand that is expected without the 
automation; this simulated manual demand is combined with the observed manual 
demand to create a pre-implementation data set. To validate this simulation, the 
automatic workload is also simulated using the same technique. The construction of the 
pre-automation data facilitates analysis of the effectiveness of the automation 
implementation at NMCSD conducted in Chapters IV and V. 
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IV. METHODOLOGY AND ANALYSIS 


The objective of this study is to quantify any efficiencies experienced due to the 
implementation of pharmacy automation at NMCSD. In addition, linear regression is 
used to evaluate which characteristics of the pre-automation and post-automation data 
sets most heavily impact the difference in average prescription fill time. The results of 
this analysis provide a foundation for the implementation of pharmacy automation at 
similarly sized medical facilities. 

A. MANUAL VS. AUTOMATIC FILLS IN CY20I4 

Using the observed post-automation data set, the prescription fill times for manual 
prescription fills and automatic prescription fills are compared; see Figure 19. In order to 
remove any evidence of autocorrelation, the observations are aggregated by week. A two- 
sided runs test for detecting non-randomness in data is performed, and the results indicate 
there is no evidence to suggest the data points were produced in a non-random manner 
(standardized runs statistic = -0.58, p-value = 0.56). On average, prescriptions filled 
automatically are completed 6.97+0.97 (standard error) minutes quicker than 
prescriptions filled manually. In CY20I4, automatic fills were faster than manual fills in 
96.2 percent of weeks, with an approximate 95 percent confidence interval for the 
probability that the automatic fills are quicker than manual fills of (0.87, 0.99). This 
confidence interval does not include 0.5, which is the expected probability if this result 
occurred by chance; therefore, this result is statistically significant. 
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Figure 19. Observed Manual and Automatic Fills in the Post-Automation Data 
Set. The average weekly prescription fill time for medications filled 
automatically tend to be quicker than manually in 96.2 percent of 
weeks in CY2014. 

It is also of interest to see if there is a significant difference in the average fill 
time during periods of high demand in the pharmacy. A period of high demand is defined 
as an hour during business hours (0800 to 2100 Monday through Friday and 0800 to 1800 
Saturday and Sunday) in which customer demand exceeds the mean yearly demand 
during that time period; the mean demand on weekdays is 83 prescriptions per hour, and 
the mean demand on weekends is 34 prescriptions per hour. These observations are 
aggregated by day, and the average daily prescription fill times during hours of high 
demand is illustrated in Figure 20. A two-sided runs test for detecting non-randomness is 
performed, and the results indicate there is no evidence to suggest dependence in the 
differences over time (standardized runs statistic = -1.49, p-value = 0.14). During periods 
of high demand, automatic fills were quicker than manual fills on 84.5 percent of days 
with an approximate 95 percent confidence interval for the probability that automatic fills 
are quicker than manual fills during hours of high demand of [0.80, 0.87]. This 
confidence interval does not include 0.5, which is the expected probability if this result 
occurred by chance; therefore, this result is statistically significant. 
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Figure 20. Manual and Automatic Fills During Flours of High Demand. During 
periods of high demand, automatic fills tend to be quicker than 
manual fills 84.5 percent of days in CY2014. 


B. PRE-AUTOMATION VS. POST-AUTOMATION 

Using both the pre-automation data set generated in Chapter III and the observed 
post-automation data set, the average daily prescription fill times pre-automation, 
YoaiiyAygPre ’ comparcd with the average daily prescription fill times post-automation, 

YDaiiyAvgPoso Figure 21. On average, prescriptions filled after the installation of 

automation are completed 4.4 ±0.34 minutes faster than prescriptions filled prior to the 
installation of automation. The average daily prescription fill time post-automation is 
faster than the average daily prescription fill time pre-automation on 92.6 percent of day 
with an approximate 95 percent confidence interval for the probability that post¬ 
automation fills are quicker than pre-automation fills of [0.89, 0.95]. This confidence 
interval does not include 0.5, which is the expected probability if this result occurred by 
chance; therefore, this result is statistically significant. 
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Figure 21. Average Daily Prescription Fill Time Pre-Automation and Post- 

Automation. The average daily prescription fill time for medications 
filled post-automation is quicker than medications filled pre¬ 
automation on 92.6 percent of days. 


Next, linear regression is used to study which characteristics of the data most 
heavily impact the difference in and This difference is the new 

response variable, defined in Equation (5). A negative value of represents a 

day in which the average prescription fill time pre-automation is quicker than post¬ 
automation. 


Y -Y -Y 151 

Equation (6) gives the linear regression model, where F is the response variable, 
Yp^.g, given Xj,...,x„ regressors; the constants, through are the unknown 

parameters, and s is the error term. The errors are modeled as independent and normally 
distributed with a mean of zero and constant variance (Farraway, 2015). Several different 
characteristics, including daily proportion of orders automated, proportion of highly 
prescribed medications, and number of daily high demand hours, are used as regressors. 
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In this analysis, the entire month of February is eliminated from consideration due to its 
relative extremes. 


^ Pl^l + ••• + Pn^n + ^ 


( 6 ) 


1. Variable Selection 

In total, ten variables are considered for impact on : total number of hours in 
the day above the mean demand proportion of daily demand fill by 

automation in the post-automation data set proportion of medications 

demanded in the top five most prescribed medications pre-automation and post¬ 
automation and respectively), proportion of medications 

demanded in the top ten most prescribed medications pre-automation and post¬ 
automation and respectively), proportion of medications 

demanded in the top 25 most prescribed medications pre-automation and post-automation 
( ^PropTop 25 MedsPre ^nd X p^^^p^^^sMedsPos, ’ rcspectively), day of the week (), and month ( 
Xu^nth)- Summary statistics for the ten considered variables are shown in Table 5. 
Variable selection is performed using backwards elimination; the full model with all ten 
variables and the final model after variable selection is validated using regression 
diagnostic techniques, and it is found that the error terms can be modeled as independent 
with mean zero and constant variance. There is some indication that the errors may not 
be normally distributed, but with the large number of observations, inference results are 
not affected (Farraway, 2015). In addition, none of the outlying observations are found to 
be influential. Diagnostic plots can be found in Appendix D. Prior to the discussion of the 
final model in Section 2, univariate analysis on the selected variables is performed. 
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Table 5. Summary Statistics for Considered Variables. The selected model 
contains three continuous variables. The coefficient estimates and 
regressor summary statistics for the regressors contained in the 
final model can be found in Table 6. 


Variable 

Final 

Type 

Minimum 

Maximum 

Mean 

Standard 

Model 

Value 

Value 

Deviation 

X 

HighDemand 

X 

Continuous 

0 

10 

6.8 

2 

X 

PropAutoPost 

X 

Continuous 

0.11 

0.48 

0.38 

0.04 

^ PropTopSMedsPre 

X 

Continuous 

0.11 

0.29 

0.17 

0.03 

^ PropTopSMedsPost 


Continuous 

0.08 

0.28 

0.16 

0.03 

^ PropTopi OMedsPre 


Continuous 

0.14 

0.35 

0.21 

0.04 

^PropTopI OMedsPost 


Continuous 

0.16 

0.41 

0.23 

0.05 

^ PropTop25MedsPre 


Continuous 

0.24 

0.49 

0.32 

0.05 

^ PropTop25MedsPost 


Continuous 

0.25 

0.53 

0.34 

0.05 

^DOW 


Categorical 

7 Levels 

N/A 

N/A 

^Month 


Categorical 

11 Levels (Excl. Feb) 

N/A 

N/A 


a. Hours of High Demand 

As defined in Section A, an hour of high demand is an hour during business hours 
in which the prescription demand is greater than the mean hourly prescription demand for 
weekends and weekdays, as appropriate. This means that instead of customers arriving to 
the pharmacy at a relatively steady pace, there are hours of peak demand. For each day, 
the number of hours the pharmacy functioned in high demand is totaled, and this 
becomes 13.1 percent of days experienced 8 or more hours of high demand. 

Figure 22 reflects a slightly decreasing trend, indicating that as the hours of high demand 
increase, the difference in the pre-automation prescription fill time and the post¬ 
automation prescription fill time decreases. 
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Figure 22. Dijference in Average Daily Prescription Fill Times Based Hours of 
High Demand. As the number of hours of high demand increases, the 
difference in average daily prescription fill time between pre¬ 
automation and post-automation decreases. 


b. Proportion Filled by Automation 

On average, automation filled 37.3 percent of prescriptions in CY2014; however, 
there are days in which automation accounted for significantly more of the total demand, 
47.7 percent, or significantly less than the total demand, 11.3 percent. In addition, 
prescriptions filled automatically have a mean fill time that is 6.94 minutes faster than 
prescriptions filled automatically. The proportion of post-automation medications filled 
through automation, , is calculated for each day. The linear fit of 

shown in Figure 23 reflects a slightly increasing trend, indicating that a higher proportion 
of prescriptions automated causes an increase in . 
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Figure 23. Difference in Average Daily Prescription Fill Times Based Daily 

Proportion of Prescriptions Automated, Post-Automation. The linear 
fit reflects a slightly increasing trend, indicating that a higher 
proportion of prescriptions automated increases the difference in 
average daily prescription fill time. 

c. Frequently Prescribed Medications 

As noted in Chapter III, the type of medication ordered influences the prescription 
fill time. The distribution for the 25 most frequently prescribed medications can be found 
in Appendix B. These medications account for 30.5 percent of total prescriptions 
filled pre-automation, and 31.7 percent of total prescriptions filled post-automation. For 
both the pre-automation and post-automation data, the daily proportion of medications 
that are the top 25, ten, and five most prescribed medications is found to create the 

variables -^/.ropropJMet/iPojr’ ^PmpTopIOMedsPre^ ^ PropTopIOMedsPost ^ ^ PropToplSMedsPre ^ 

^PropTop 25 MedsPost ' Thcsc varlablcs are all highly correlated with each other; therefore, it is 

only appropriate to select one of them for use in the linear regression. After performing 
backwards elimination, is selected as the variable with the most impact on 

Yjj-f .. This is a reasonable selection because frequently prescribed medications are stored 
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in more convenient locations, and therefore, their fill times should be faster than less 
frequently prescribed medications. When all prescriptions are filled manually, this 
convenience could make a large difference in the overall daily prescription fill time. The 
linear fit of shown in Figure 24 reflects a slightly increasing trend, 

indicating that a higher proportion of prescriptions automated causes an increase in . 

Linear fits of ’ ^PmpTopWMedsPre ’ ^PropToplOMedsPost ’ ^PropTop25MedsPre ’ 


^ PropTop25MedsPost 


can be found in Appendix E. 
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Figure 24. Dijference in Average Daily Prescription Fill Times Based Daily 
Proportion of Prescriptions that are the Five Most Prescribed 
Medications, Pre-Automation. The linear fit reflects a slightly 
increasing trend, indicating that a higher proportion of highly 
prescribed medications increases the difference in average daily 
prescription fill time. 


2. Regression 

A simple linear regression is fit using the three selected continuous variables: 
^HighDe^and^ ^PropAutoPoso ^nd X. Equatiou (7) shows the resulting model, where 


each estimate of the coefficients are rounded to the corresponding standard error. A 
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positive value of the response signifies that the post-automation prescription fills were, 
on average, faster than the pre-automation prescription fills on that particular day. A 
positive (negative) value of a coefficient indicates that as the variable increases, the 
difference in the average daily prescription fill time between pre-automation and post¬ 
automation becomes more positive (negative). Additional details for the regressors can be 
found in Table 6. 


Yoiff ■'^^HoursHigh ^^PwpAutoPost PwpTopSPre 


(7) 


Table 6. Model Term Coefficient Estimates, Standard Error, T-Ratio, and 
Statistical Significance. Each coefficient is rounded in the final 
regression model according to the corresponding standard error. 
Each variable is significant to the 0.1 level. 


Term 

Estimate 

Standard 

Error 

T-Ratio 

P-Value 

Intercept 

0.55 

2.53 

-0.22 

0.8265 

X 

HighDemand 

-0.25 

0.10 

-2.45 

0.0150 

X 

PropAutoPost 

8.86 

4.79 

1.85 

0.0651 

^ PropTop5Pre 

12.30 

6.74 

1.83 

0.0688 


This linear regression model has three degrees of freedom with 333 residual 
degrees of freedom. These values correspond to an E-ratio of 6.870, and a model p-value 
of 0.0002; this result indicates there is a significant relationship between the variables in 

the linear regression to the response. The value, which is a measure of the proportion 

of the variability of the response explained by the regressors, is 0.058 (Earraway, 2015). 
Though this is an extremely low value for this measure, it is not inherently limiting. The 
regressors are still statistically significant, and conclusions can still be drawn about how 
changes in the regressors are associated with changes in the response. The intention of 
this study is not to predict the difference in average daily prescription fill times for pre¬ 
automation and post-automation, but to determine the factors that impact this response. 
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The impact of these three regressors on the difference in average daily 
prescription fill time pre-automation and post-automation can be inferred from the values 
of their coefficients. The value of the coefficient for suggests that for every 

increase in hours of high demand experienced by the pharmacy, the estimated expected 
response decreases by 0.2 minutes, holding the values of the other two regressors 
constant. The maximum value of the variable found in the data is ten hours; therefore, 
this variable could potentially decrease the response by two minutes. A decrease in the 
response indicates that the implementation of automation had a lesser impact on the 
overall average daily prescription fill time. 

The proportion of prescriptions filled through automation, causes an 

increase in response. This variable can take on values between zero and one, and the 
maximum value of , d observed in the data set is 0.48; this indicates that 

Xp^opAutoPost could potentially increase the response by 3.84 minutes, holding the values of 

the other two regressors constant. An increase in the response indicates that the 
implementation of automation has a greater impact on the overall daily prescription fill 
time. Figure 25 depicts the change in the response for different values of 

fixed values of . 
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Figure 25. Estimated Expected Difference in Average Daily Prescription Eill 
Time as Hours of High Demand Increases for Eixed Values of 
Proportion of Prescriptions Automated. The proportion of automated 
prescription fills is fixed at the first quartile and maximum values, 
while the hours of higher than average demand are varied. As the 
hours of higher than average demand increase, the estimated 
expected difference in average daily prescription fill time decreases. 
The dotted lines represent 95 percent confidence intervals. 


The proportion of highly prescribed medications pre-automation, , also 

positively impacts This variable takes on values between zero and one and the 
maximum value of x„ ^ observed in the data set is 0.29. Based on this information, 
an increase in could potentially increase the difference in average daily 

prescription fill times by 3.48 minutes, holding the values of the other two regressors 
constant. This increase is an interesting result, because frequently ordered medications 
tend to be stored in more convenient locations; it is expected that this would cause a 
decrease in . The observed result, an increase, could be explained by an optimized 

loading of medications into the automation machine. Figure 26 depicts the change in the 
response for different values of fixed values of Xp^^^j^^^p^^. 
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Figure 26. Estimated Expected Difference in Average Daily Prescription Eill 
Time as Hours of High Demand Increases for Eixed Values of 
Proportion of Erequently Prescribed Medications. The proportion of 
frequently prescribed medications is fixed at the first quartile and 
maximum values, while the hours of higher than average demand are 
varied. As the hours of higher than average demand increase, the 
estimated expected difference in average daily prescription fill time 
decreases. The dotted lines represent 95 percent confidence intervals. 


C. SUMMARY 

Through analysis of the data, it is revealed that the difference in average daily 
prescription fill times is statistically significant, both between manual and automatic fills 
in CY2014, and between pre-automation and post automation. In addition, linear 
regression reveals that there are three main factors that affect the difference in average 
daily prescription fill times: the number of hours of high demand experienced within the 
pharmacy, the proportion of prescriptions filled by automation, and the proportion of 
highly prescribed medications demanded by customers. As the number of hours of high 
demand increases, the response decreases, while as the two proportions increase, the 
response also increases. 
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V. RESULTS, CONCLUSIONS, AND RECOMMENDATIONS 


This thesis demonstrates a method to evaluate the changes in efficiency 
experienced at the NMCSD pharmacy after the implementation of pharmacy automation. 
The introduction of barcode scanning for each step in the prescription fill process 
provides a unique opportunity to objectively evaluate this process. With the increase in 
emerging technologies now available throughout different areas of healthcare, it is in the 
Navy’s best interest to invest in items that could potentially lead to better access to care; 
this, in turn, leads to healthier Sailors and Marines. 

A. POST-AUTOMATION TRENDS 

Several items of interest emerged during exploratory analysis of the post¬ 
automation data set. With more than one year’s worth of data, these trends could be 
further studied to determine if they are reoccurring, or specifically related to events in 
CY2014. The most significant and problematic item is the sharp increase experienced in 
the month of February. During this month, the average prescription fill time during 
working hours on weekdays is 63.3 ± 162.2 (standard deviation) minutes. In comparison, 
the average prescription fill time for the rest of the year during this time same time period 
is 24.3 ±55.9 minutes. The month of February also experiences an increase in average 
daily demand of 197 prescriptions compared to every other month in the year. It can be 
hypothesized that this departure from the norm is associated with the cold and flu season 
and an increase in patient visits to NMCSD; if is it determined that this increase in 
demand and prescription fill time is not an anomaly, extra pharmacists and pharmacy 
technicians should be staffed during this month to account for the increase in demand. 

Next, data is aggregated by day of the week. There is an obvious difference in 
both demand and average prescription fill time between weekdays and weekends; this is 
expected based on the NMCSD’s primarily military patient population. A more 
interesting observation is the average prescription fill time is significantly higher on 
Mondays than every other day of the week, but that there is no corresponding increase in 
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demand. During working hours, the average time to fill a prescription on Mondays is 
34.9 ± 108.1 minutes, compared to every other weekday, which is 26.1 ±61.6 minutes 

Again, much of this increase can be attributed to the volatile month of February. 
Excluding February, the average hourly prescription fill time during working hours on 
Mondays is 26.8 + 66.4 minutes, while in February alone, the average hourly prescription 
fill time on Mondays is 114.6 + 275.5 minutes. Figure 27 compares the hourly 
prescription fill times in February to the rest of the year. While the times are considerably 
higher during February, there is still a substantial difference in the fill times from 1800- 
1900 during the rest of the year. Excluding February, the average prescription fill time 
during this hour on Mondays is 81.6 + 267.9 minutes. This increase in prescription fill 
time during 1800-1900 is not related to an increase in demand; the average demand 
during this hour is 40 prescriptions, compared to average of 70 items per hour during the 
rest of the day. Prescription fill time only includes the time it takes to get the prescription 
to the pharmacist for verification, which means a customer could potentially be waiting 
upwards of 90 minutes for their specific order. 



Figure 27. Average Hourly Prescription Fill Time During Working Hours, 

Monday, February Only and Excluding February. The black bars 
represent standard error of the mean. Though Mondays in February 
represent a considerable departure from the average, the average 
prescription fill time in the hour from 1800 to 1900 is significantly 
higher than the rest of the workday, without a corresponding increase 
in demand. 
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This increase in prescription fill time during the latter part of the day is not just 
limited to Mondays, however. If the entire year is observed, there is an increase in 
average prescription fill time during the hour from 1800 to 1900, but the demand is 
almost at a daily low. This increase in prescription fill time is also not attributable to 
variance in manual fill times; Figure 28 displays the average prescription fill time, 
excluding February, for weekdays between 0700 and 2200, separated by type of fill. 
Average manual fill times from 1800 to 1900 are 34.4±98.8 with a corresponding 
average hourly demand of only 18 items, while average automatic fill times during the 
same hour are 29.9 ±79.8 minutes with a corresponding average hourly demand of 13 
items. In comparison, for the rest of the working day, an average manual fill takes 26.7 ± 
55.2 minutes with an average demand of 37 prescriptions; an average automatic fill is 
completed in 19.3 ±39.8 minutes with an average demand of 21.3 prescriptions. 



Figure 28. Average Hourly Prescription Demand and Prescription Fill Time, 
Working Hours, Monday through Friday, excluding February. The 
black bars represent standard error of the mean. Though the demand 
and average prescription fill time for manual fills is higher than 
automatic fills for all hours during the work week, both manual and 
automatic fills experience a delay from 1800 to 1900 without a 
corresponding increase in prescription demand. 
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Without more information, it is difficult to determine the source of these delays. 
The absence of a corresponding increase in demand indicates this increase in prescription 
fill time in unrelated to an influx of customers into the pharmacy. Both the manual 
prescription fill times and the automatic prescription fill times increase significantly 
during this hour; there is a 28.8 percent increase in average hourly prescription fill times 
for manual fills and a 51.8 percent increase for automatic fills. This evidence suggests the 
delays are related to workflow management within the pharmacy. 

B. PRE-AUTOMATION COMPARED TO POST-AUTOMATION 

When evaluating the total time spent filling prescriptions both pre-automation and 
post-automation, it is important to understand that several technicians are filling 
prescriptions at the same time; in the post-automation environment, the automation 
system is also continuously routing prescriptions through the fill process. The total time 
spent filling prescriptions reflects the amount of time it would take to fill each 
prescription sequentially, one at a time. In most instances, it is most appropriate to refer 
to total prescription fill time for the year in the units of days. 

During regression analysis, one of the significant indicators of a larger difference 
between the average daily prescription fill time pre-automation and post-automation is 
the post-automation proportion of prescriptions filled automatically. Overall, the 
prescriptions filled automatically account for 37.3 percent of total prescription fills in 
CY2014, with a maximum of 47.7 percent and a minimum of 11.2 percent. With all 
other factors held constant, an average automation proportion of 0.37 is related to an 
increase in the difference of average prescription fill time of 2.98 ±1.95 (standard error) 
minutes; if this proportion can be maintained over the course of an entire year, assuming 
an average daily weekday demand is 1186 prescriptions, this could reduce yearly 
workload by an average of 636 days alone, compared to pre-automation. Figure 29 
illustrates the yearly reduction in weekday workload based on proportion of prescriptions 
filled through automation. 


52 




Figure 29. Estimated Yearly Reduction in Weekday Workload Based on 
Proportion of Prescriptions Filled Through Automation and 
Demand. If the proportion of preseriptions automated can be 
maintained at 0.37, there is potential to reduce weekday workload 
within the pharmacy by 636 days, compared to pre-automation. 

With all other regressors held constant, there is no additional 
reduction of work in any scenario when the proportion of workload 
automated is set to zero, causing a non-linear line. 

This same type of logic can be applied to the daily workload of each pharmacy 
technician in NMCSD. Based on the January 2015 schedule, at a minimum, 25 pharmacy 
technicians are assigned to work in the pharmacy from 0745 to 1615 every weekday, with 
other additional staff during various parts of the workday; less pharmacy technicians are 
assigned during non-working hours (B. Detlef, BUMED, personal communication, 
August 12, 2015). Assuming there are 25 pharmacy technicians working within the 
pharmacy, if the proportion of prescriptions automated is held constant at 0.37 for an 
entire day, with an average weekday demand of 1,186 items, the daily workload for each 
technician can be reduced by 2.34 + 0.03 hours, compared to pre-automation. When this 
proportion is maintained at 0.50, the daily workload can be reduced by 3.16 ±0.02 hours, 

which is a 0.40 Full Time Equivalent (FTE), assuming an eight-hour workday. Eigure 30 
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illustrates this reduction in daily workload based on proportion of prescriptions 
automated and number of pharmacy technicians working in the pharmacy. 



Figure 30. Estimated Daily Reduction in Weekday Workload Based on 

Proportion of Prescriptions Filled Through Automation and Number 
of Pharmacy Technicians. If the proportion of prescriptions 
automated can be maintained at 0.37, with ten pharmacy technicians 
working and an average demand of 1,189 items, there potential to 
reduce daily workload by 2.34 hours, which is the same as 0.29 
FTEs. With all other regressors held constant, there is no additional 
reduction of work in any scenario when the proportion of workload 
automated is set to zero, causing a non-linear line. 


It is also important to consider the implementation from a cost perspective. The 
pharmacy automation system at NMSCD carried an individual implementation cost of 
$2.4 million of the total $49 million contract. (B. Detlef, BUMED, personal 
communication, September 10, 2014). Based on the watchbill from January 2015, most 
of the pharmacy technicians are of the rank Petty Officer Third Class (E-4), so the base 
salary for this rank, with over three years of service, will be used at the basis for cost 
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analysis (B. Detlef, BUMED, personal communication, August 12, 2014); there is a mix 
of several military ranks, as well as civilians and contractors, which suggests that this 
estimate is conservative. Appendix F details the conversion of E-4 military salary to 
hourly wage. 

When directly comparing the total time spent filling prescriptions pre-automation 
and post-automation, recall the yearly demand for each is roughly the same: 364,463 and 
364,531 observations, respectively. Pre-automation, the total time spent filling 
prescriptions is 7,699 days, while the total time spent filling prescriptions post 
automation is 6,604 days; this is a reduction of 14.2 percent. This reduction in time spent 
filling prescriptions is associated with $330,000 reduction in cost, the amount of money it 
would cost to pay for the equivalent amount of work from E-4 military members 
established only on base pay. With these calculations, it would take roughly seven years 
to recoup the cost of implementation at NMCSD based on increased efficiency. 

These are extremely conservative estimates. In reality, many other factors 
contribute to the benefits of implementing a new system. For example, the increased 
tracking ability due to the barcode scanning of medications could potentially lead to 
increased efficiency in the stocking and ordering of medications. In addition, wait times 
in the pharmacy are a major source of complaints for many customers within a medical 
facility, and the main reason many customers choose to have prescriptions filled at non¬ 
military pharmacies. Prescriptions filled at non-military pharmacies are more expensive 
for the Navy to fill, and the potential decrease in customer wait time within the pharmacy 
could then be tied to a recapture of costs from non-military pharmacy prescriiption fills. 
Decreased wait time also likely leads to better patient satisfaction, which is an intangible 
benefit of such an upgrade. 

C. CONCLUSIONS AND FUTURE WORK 

After the implementation of pharmacy automation at NMCSD, there is an 
increase in efficiency, defined as the total prescription fill time. This efficiency can be 
quantified as number of reduction total time to fill a prescription, reduction in FTEs, or 
reduction in equivalent salary of pharmacy technicians. This data is collected after a little 
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more than one year of implementation; with better training and more emphasis placed on 
the optimal automation workload, these savings in efficiency will continue to increase. 

There is great potential for future study into this subject. With the 
implementation of the barcode scanning system, more in-depth investigations can be 
made into the optimal ordering and stocking of frequently filled medications. In addition, 
this study does not attempt to quantify the impacts of the automation system on 
prescription accuracy, patient satisfaction, or overall access to care. With additional data, 
comparisons can be made between changes in the pharmacy to increases or decreases in 
these areas with all of NMCSD. Most importantly, this study is limited by lack of data, 
both in duration of post-automation data collection and lack of pre-automation data; with 
additional post-automation data, a more significant evaluation of possible trends observed 
within the pharmacy could be conducted, and with pre-automation data, a considerably 
better comparison could be completed. 
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APPENDIX A. PROCESSES TRACKED IN PRESCRIPTION DATA 


Table 7. Processes Tracked in Prescription Data. The prescription data 
contains the date and time, accurate to the second, of fifteen 
specific processes during the entire prescription order procedure. 
Priority High, Priority Urgent, Initial Pending, Suspend, 

Reschedule to Manual, and Rescheduled to Automatic do not occur 
with every prescription; these processes tend to happen when there 
is a complication. Only the eight processes outline in Chapter III 
are specifically relevant to this study. 


Process 

Description (date/time) 

Scheduled 

Entered into the system 

Diverted 

Divert from an automatic fill to manual fill, reason provided 

Priority High 

Priority is updated from standard to high 

Priority Urgent 

Priority is updated from high to urgent 

Initial Pending 

Initially put into a “pending” status, reason provided 

Suspend 

Suspended from completion, reason provided 

Rescheduled to Manual 

Rescheduled from an automatic fill to manual fill 

Rescheduled to Automatic 

Rescheduled from manual fill to automatic fill 

Counted 

Count of quantity of medication is complete 

Added to Tote 

Counting, bottling, capping, and labeling is complete 

Verified 

Pharmacist verification for errors is complete 

Added to Bag 

Combined with remaining customer order 

Added to Location 

Complete order is placed in assigned cubby location 

Ready for Pickup 

Order is complete and ready for customer pick up 

Pickup 

Customer picks up completed order. 
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APPENDIX B. FREQUENCY OF PRESCRIBED MEDICATIONS 



Figure 31. Most Frequently Prescribed Medications, Post Automation. The 25 
most frequently preseribed medieations comprise 31.7 percent of all 
post-automation prescription fills. 
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Figure 32. Most Frequently Prescribed Medications, Pre-Automation. The 25 
most frequently preseribed medieations eomprise 30.5 pereent of all 
pre-automation prescription fills 
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APPENDIX C. DIFFERENCE IN AVERAGE FILL TIME IN 
OBSERVED AND SIMULATED DATA SET, MONTH BY DAY OF 

THE WEEK. 



Figure 33. Month by Day of the Week for the Difference in Average Daily 
Prescription Fill Time between the Observed and Simulated 
Automation Data Set, First Quadrimester CY2014. Other than the 
month of February, there does not appear to be any significant 
patterns based on month or day of week 
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Figure 34. Month by Day of the Week for the Difference in Average Daily 
Prescription Fill Time between the Observed and Simulated 
Automation Data Set, Second Quadrimester CY2014. There does not 
appear to be any significant patterns based on month or day of week 
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Figure 35. Month by Day of the Week for the Difference in Average Daily 
Prescription Fill Time between the Observed and Simulated 
Automation Data Set, Third Quadrimester CY2014. There does not 
appear to be any significant patterns based on month or day of week 
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APPENDIX D. REGRESSION DIAGNOSTICS 



Figure 36. Regression Diagnostic Plots for Final Linear Regression Model. The 
final model is validated using regression diagnostics. The four 
residual plots indicate that the error terms have constant variance and 
mean zero, but may not be normally distributed. In addition, none of 
the outlying observations are found to be influential. 
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APPENDIX E. LINEAR EITS OE ELIMINATED VARIABLES 



'■j 

•j 

c 

a 

.'j 

Q 


20 - 

10 - 

I _ L i 1 i 

1 1 1 


0 

I 1 I r 


! 1 

r 


• 


• 



i 


-10 - 

• 



-20 - 

• 



-30 - 


Linear Fit 



• 

- 95 Percent Contiilence Interval 


-40 - 






Sun Mon Tuc Wed Thu Fri Sat 


Figure 37. Difference in Average Daily Prescription Fill Times Based on Day 
of the Week. The linear fit reflects a straight line, indicating that day 
of the week does not impact the difference in average daily 
prescription fill time. 
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Figure 38. Difference in Average Daily Prescription Fill Times Based on 

Month. The linear fit reflects a straight line, indicating that month 
does not impact the difference in average daily prescription fill time. 
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Figure 39. Difference in Average Daily Prescription Fill Times Based Daily 
Proportion of Prescriptions that are the Five Most Prescribed 
Medications, Post-Automation. The linear fit refleets a slightly 
increasing trend, indicating that a higher proportion of highly 
prescribed medications increases the difference in average daily 
prescription fill time. 
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Figure 40. Difference in Average Daily Prescription Fill Times Based Daily 
Proportion of Prescriptions that are the Ten Most Prescribed 
Medications, Pre-Automation. The linear fit reflects a slightly 
increasing trend, indicating that a higher proportion of highly 
prescribed medications increases the difference in average daily 
prescription fill time. 
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Figure 41. Dijference in Average Daily Prescription Fill Times Based Daily 
Proportion of Prescriptions that are the Ten Most Prescribed 
Medications, Post-Automation. The linear fit reflects a slightly 
increasing trend, indicating that a higher proportion of highly 
prescribed medications increases the difference in average daily 
prescription fill time. 
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Figure 42. Difference in Average Daily Prescription Fill Times Based Daily 
Proportion of Prescriptions that are the 25 Most Prescribed 
Medications, Pre-Automation. The linear fit reflects a slightly 
increasing trend, indicating that a higher proportion of highly 
prescribed medications increases the difference in average daily 
prescription fill time. 
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Figure 43. Dijference in Average Daily Prescription Fill Times Based Daily 
Proportion of Prescriptions that are the 25 Most Prescribed 
Medications, Post-Automation. The linear fit reflects a slightly 
increasing trend, indicating that a higher proportion of highly 
prescribed medications increases the difference in average daily 
prescription fill time. 
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APPENDIX F. CONVERSION OF MONTHLY EARNINGS TO 

HOURLY WAGES 


Military salary is normally presented in terms of monthly earnings. Equation (8) 
details the conversion of monthly earnings for an E-4 into hourly wages (Defense 
Einance and Account Service, 2015). 


$2238.07 ^ 
Month / 


12 Months^ r 1 Year ^ 
1 Year A 261 Weekdays J 


1 Weekday 3 _ $12,863 
8 Work-Hours/ v Hour / 


( 8 ) 
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